Image encoding apparatus, image decoding apparatus and image transmission method

ABSTRACT

According to an embodiment, an image encoding apparatus includes a background control information generator, a background image generator, and an image encoder. The background control information generator generates background control information based on an input image. The background control information is used for generating a first background image from at least one first image. The background image generator generates the first background image, based on the background control information and at least one first image. The image encoder encodes the input image using the first background image to generate encoded data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2014-232702, filed Nov. 17, 2014, theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to image transmission.

BACKGROUND

In general, where images (video) photographed by a camera aretransmitted to a remote plate, the images are encoded (compressed) toreduce the amount of data transmitted. Typically, a high coding rate isenabled in image encoding by referring to previously encoded images andpredicting a present image. If the images are transmitted by way of anarrow-band channel, however, the high image qualities are hard tomaintain.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating an image transmission systemaccording to the first embodiment.

FIG. 2 is a block diagram illustrating a background image generatordepicted in FIG. 1.

FIG. 3 is a block diagram illustrating a background control informationgenerator depicted in FIG. 1.

FIG. 4 is a block diagram illustrating an image encoder depicted in FIG.1.

FIG. 5 is a block diagram illustrating an image decoder depicted in FIG.1.

FIG. 6 is a block diagram illustrating an image transmission systemaccording to the second embodiment.

FIG. 7 is a block diagram illustrating a background control informationgenerator depicted in FIG. 6.

FIG. 8 is a block diagram illustrating an image transmission systemaccording to the third embodiment.

FIG. 9 is a block diagram illustrating a background control informationgenerator depicted in FIG. 8.

FIG. 10 is a block diagram illustrating an image transmission systemaccording to the fourth embodiment.

FIG. 11 is a block diagram illustrating a background image generatordepicted in FIG. 10.

FIG. 12 is a block diagram illustrating a background control informationgenerator depicted in FIG. 10.

FIG. 13 illustrates an example of hardware that realizes the imageencoding apparatus, image decoding apparatus and background controlinformation generation apparatus included in the image transmissionsystem according to each embodiment.

DETAILED DESCRIPTION

A description will now be given of the embodiments with reference to theaccompanying drawings.

According to an embodiment, an image encoding apparatus includes abackground control information generator, a background image generator,and an image encoder. The background control information generatorgenerates background control information based on an input image. Thebackground control information is used for generating a first backgroundimage from at least one first image. The background image generatorgenerates the first background image, based on the background controlinformation and at least one first image. The image encoder encodes theinput image using the first background image to generate encoded data.

According to an image decoding apparatus includes a background controlinformation generator, a background image generator and an imagedecoder. The background control information generator generatesbackground control information based on a previously decoded image. Thebackground control information is used for generating a background imagefrom at least one image. The background image generator generates thebackground image, based on the background control information and atleast one image. The image decoder decodes encoded data using thebackground image to generate a reproduced image.

In the descriptions below, the same reference numerals or symbols willbe used to refer to already-described elements or elements similar tothem, and redundant descriptions will be omitted.

First Embodiment

As shown in FIG. 1, the image transmission system of the firstembodiment comprises an image encoding apparatus 100 and an imagedecoding apparatus 200. The image encoding apparatus 100 includes animage encoder 110, a background image generator 120 and a backgroundcontrol information generator 130. The image decoding apparatus 200includes an image decoder 210 and a background image generator 220. Theimage transmission system shown in FIG. 1 may be, for example, a systemthat transmits images photographed by a monitor camera to a remoteoperator terminal.

The image encoder 110 receives an input image (signal) 10, and alsoreceives a background image (signal) 14, to be mentioned later, from abackground image generator 120. The image encoder 110 encodes the inputimage 10 using the background image 14, thereby generating encoded data15. The image encoder 110 transmits the encoded data 15 to the imagedecoding apparatus 200. The encoded data 15 may be transmitted to theimage decoding apparatus 200 by wireless communications or suppliedthereto by wire communications; alternatively, the encoded data 13 maybe transferred to the image decoding apparatus 200, using a recordingmedium.

The encoded data 15 can have a high coding rate by performing predictiveencoding using the background image 14. For example, if the backgroundimage 14 is used as a predictive image, most of the background regionincluded in the input image 10 does not have to be encoded ortransmitted. As a result, the amount of encoded data 15 can besignificantly reduced.

FIG. 4 shows a specific example of the image encoder 110. The imageencoder 110 shown in FIG. 4 includes a subtractor 301, atransformer/quantizer 302, a de-quantizer/inverse transformer 303, anadder 304, a loop filter 305, a decoded image buffer 306, an externalimage buffer 307, a predicted image generator 308, an entropy encoder309 and an encoding controller 310.

The subtractor 301 receives an input image 10, and also receives apredicted image from the predicted image generator 308. The subtractor301 generates a prediction error (signal) by subtracting the predictedimage from the input image 10. The subtractor 301 supplies theprediction error to the transformer/quantizer 302.

The transformer/quantizer 302 generates transform coefficients bytransforming the prediction error, and generates quantized transformcoefficients by quantizing the transform coefficients in accordance witha preset quantization parameter. Orthogonal transforms, such as adiscrete cosine transform (DCT), are used for the transforming theprediction error. The transformer/quantizer 302 supplies the quantizedtransform coefficients to the de-quantizer/inverse transformer 303 andthe entropy encoder 309.

The de-quantizer/inverse transformer 303 receives the quantizedtransform coefficients from the transformer/quantizer 302. Thede-quantizer/inverse transformer 303 de-quantizes the quantizedtransform coefficients in accordance with the quantization parametermentioned above, thereby restoring the transform coefficients to theoriginal state, and performs inverse transform with respect to thetransform coefficients (the inverse transform is processing just theopposite to the transform performed by the transformer/quantizer 302),thereby restoring the prediction error to the original state. Inverseorthogonal transforms, such as an inverse DCT (IDCT), are used for theinverse transforms of the transform coefficients. Thede-quantizer/inverse transformer 303 supplies the prediction error tothe adder 303.

The adder 304 receives the prediction error from thede-quantizer/inverse transformer 303, and the predicted image from thepredicted image generator 308. The adder 304 adds the prediction errorand the predicted image together, thereby generating a local decodedimage (signal). The adder 304 supplies the local decoded image to theloop filter 305.

The loop filter 305 receives the local decoded image from the adder 304.The loop filter 305 applies loop filter processing to the local decodedimage. For example, the loop filter processing may be deblocking filterprocessing. The loop filter 305 supplies the filtered local decodedimage to the decoded image buffer 306.

The decoded image buffer 306 receives the filtered local decoded imagefrom the loop filter 305. The decoded image buffer 306 stores thefiltered local decoded image as a reference image. The decoded imagebuffer 306 supplies the reference images to the predicted imagegenerator 308, as needed.

The external image buffer 307 receives the background image 14 from thebackground image generator 120. The external image buffer 307 stores thebackground image 14 as a reference image. The external image buffer 307supplies the reference images to the predicted image generator 308, asneeded.

The predicted image generator 308 acquires the input image, receives thereference image (filtered local decoded image) from the decoded imagebuffer 306, receives the reference image (background image 14) from theexternal image buffer 307, and receives prediction control informationfrom the encoding controller 310. The predicted image generator 308generates a predicted image, using both the input image 10 and thereference images and in accordance with the prediction controlinformation. For example, the predicted image generator 308 generates apredicted image by performing intra prediction, motion compensationprediction for the reference images, or prediction from the backgroundimage 14 (corresponding to the inter-layer prediction in hierarchicalcoding). The predicted image generator 308 supplies the predicted imageto both the subtractor 301 and the adder 304. In addition, the predictedimage generator 308 supplies prediction information regarding thepredicted image (including motion vector information) to the entropyencoder 309.

The entropy encoder 309 receives the quantized transform coefficientsfrom transformer/quantizer 302, and the prediction information from thepredicted image generator 308. The entropy encoder 309 performs entropyencoding for both the quantized transform coefficients and theprediction information in accordance with the syntax, thereby generatingencoded data 15. The entropy encoder 309 supplies the encoded data 15 toan apparatus external to the image encoder 110. The entropy encoding is,for example, Huffman encoding, arithmetic encoding, or the like. Theencoding controller 310 generates prediction control information andsupplies this information to the predicted image generator 308.

The background image generator 120 receives a fundamental image (signal)11, and also receives background control information 13 from thebackground control information generator 130. The background imagegenerator 120 generates a background image 14 based on the fundamentalimage 11 and the background control information 13. The background imagegenerator 120 supplies the background image 14 to the image encoder 110.

To be specific, the fundamental image 11 may comprise one or more stillimages; alternatively, it may comprise a moving picture including aplurality of image frames. To enhance the coding rate, it is preferablethat the fundamental image 11 include images which are likely toresemble the input image in terms of the photographing position, thephotographing direction, etc. For example, if the input image 10 is animage which is photographed by a camera attached to a first movingobject (e.g., a train and a bus) when this first moving object travelsalong a predetermined route, then the fundamental image 11 may be a pastimage which is photographed by a camera attached to a second movingobject (which may be the same as, or different from the first movingobject) when the second moving object travels along the same route.Alternatively, the fundamental image 11 may be an image photographed inthe past by a fixed camera or an externally controllable camera thatphotographs the input image 10.

The background control information 13 includes a parameter required forgenerating the background image 14. To be specific, the backgroundcontrol information 13 may include selection information representing atleast one image (selection image) that is selected as the base of thebackground image 14 of the fundamental image 11. The selectioninformation may be a frame number if the fundamental image 11 is amoving picture. The background control information 13 includes acorrection parameter used for enhancing the quality of the backgroundimage 14.

For example, the background image generator 120 may generate thebackground image 14 by selecting one image (selection image) from atleast one fundamental image 11 based on the background controlinformation 13. Alternatively, the background image generator 120 maygenerate the background image 14 by selecting a plurality of images(selection images) from a plurality of fundamental images 11 based onthe background control information 13 and executing background pixelcalculation (to be described later), using the selection images. Inaddition, the background image generator 120 may execute formatconversion or image correction for the background image 14 generated asabove, before the background image 14 is supplied to the image encoder110.

If the background image 14 differs from the input image 10 in format,then the background image generator 120 converts the format of thebackground image 14 to the same format as the input image 10. The formatis, for example, an image size, an image bit depth, a color differenceformat, a color space, etc.

Where the background control information 13 includes a correctionparameter, the background image generator 120 corrects the backgroundimage 14 using the correction parameter. To be more specific, thebackground image generator 120 may execute gamma correction, using thegamma value included in the correction parameter. The background imagegenerator 120 may execute histogram conversion, using the histograminformation on the input image included in the correction parameter. Thebackground image generator 120 may execute Wiener filter processing forthe background image 14, using the filter coefficient of the Wienerfilter included in the correction parameter (the filter coefficient canbe determined based on the input image 10 and the background image 14).The background image generator 120 may execute weighted prediction ofthe background image 14, for example, for the correction of thebrightness, using the weight coefficient and offset value included inthe correction parameter. The background image generator 120 may executegeometric transform of the background image 14, for example, for thecorrection of the photographing position, using the geometric-transformparameter value included in the correction parameter. The backgroundimage generator 120 may execute a variety of image corrections notmentioned herein.

FIG. 2 shows a specific example of the background image generator 120.The background image generator 120 shown in FIG. 2 includes afundamental image buffer 121, a background pixel calculator 122 and abackground image buffer 123.

The fundamental image buffer 121 receives and stores a fundamental image11. The fundamental image 11 stored in the fundamental image buffer 121is read out by the background pixel calculator 122, as needed.

The background pixel calculator 122 receives background controlinformation 13 from the background control information generator 130.The background pixel calculator 122 specifies a plurality of selectionimages based on the background control information 13 and reads theselection images from the fundamental image buffer 121. The backgroundpixel calculator 122 generates a background image 14 (pixel valuesconstituting the background image) by executing background pixelcalculation using the selection images read out from the fundamentalimage buffer 121. The background pixel calculator 122 supplies thebackground image 14 to the background image buffer 123.

The background pixel calculation may be any kind of arithmeticprocessing for deriving background pixels from a plurality of images(selection images) that are adjusted in position and photographed atdifferent photographing times. For example, the background pixelcalculator 122 may calculate a mode of the pixel values at predeterminedpositions of the selection images, thereby deriving the pixel values ofthe predetermined position of the background image 14. The backgroundpixel calculator 122 may calculate a median or an average in place ofthe mode. The background pixel calculator 122 may determine a thresholdused for determining non-background pixels (foreground pixels), based onthe average and variance of the pixel values at predetermined positionsof a plurality of selection images. In this case, the background pixelcalculator 122 first excludes non-background pixels from the pixelvalues, and re-calculates an average to derive the pixel values at thepredetermined position of the background image 14. Alternatively, thebackground pixel calculator 122 may calculate each of the pixel valuesof the background image 14, using the Gaussian mixture model (GMM) orthe Godbehere, Matsukawa, Goldberg algorithm (GMG algorithm).

A general image transmission system uses a color space in which a pixelvalue is expressed based on one brightness signal and twocolor-difference signals. However, the brightness signal is easilyaffected by a shadow in the background pixel calculation. If the colorspace of a selection image includes a brightness signal, therefore, thebackground pixel calculator 122 may first convert the color space into acolor space including no brightness signal (e.g., RGB) and then executebackground pixel calculation. Furthermore, the background pixelcalculator 122 may change each pixel value of the background image 14back into an original color space, as needed.

The background image buffer 123 receives the background image 14 (thepixels values constituting the background image) from the backgroundpixel calculator 122, and stores the received background image 14. Thebackground image 14 stored in the background image buffer 123 is readout by the image encoder 110, as needed.

The background control information generator 130 receives an input image10, a fundamental image 11 and supplemental information 12. Thebackground control information generator 130 searches the fundamentalimage 11 for an image similar to the input image, based for example onthe supplemental information 12, and generates background controlinformation 13 including selection information representing the imagesearched for (which image is the same as the selection image describedabove). Furthermore, the background control information generator 130may generate a correction parameter used for correcting the backgroundimage 14 and include it in the background control information 13.

The supplemental information 12 may include information representing,for example, a photographing position, a photographing direction, aphotographing date, a photographing time, etc. These kinds ofinformation are useful when the background control information generator130 searches the fundamental image 11 for an image similar to the inputimage 10. For example, the information on the photographing position andphotographing direction is useful when the fundamental image 11 issearched for an image which is similar to the input image in terms of anobject (especially, a background) to be photographed. The information onthe photographing date and time is useful when the fundamental image 11is searched for an image which is similar to the input image 10 in termsof the sunshine condition.

The background control information generator 130 may search thefundamental image 11 for an image similar to the input image, withoutusing the supplemental information 12. Alternatively, the backgroundcontrol information generator 130 may select at least one image from thefundamental image 11, without reference to the input image 10. In orderto improve the coding rate, an image similar to the input image 10should be preferably searched for. In order to improve the searchefficiency, the supplemental information 12 should be preferably used.

The background control information generator 130 supplies the backgroundcontrol information 13 to the background image generator 120. Inaddition, the background control information generator 130 supplies thebackground control information 13 to the image decoding apparatus 200.The background control information 13 may be transmitted to the imagedecoding apparatus 200 by wireless communications or supplied thereto bywire communications; alternatively, the background control information13 may be transferred to the image decoding apparatus 200, using arecording medium.

The background control information 13 may be transmitted separately fromthe encoded data 15 or together with the encoded data 15. For example,the background control information 13 and the encoded data 15 aretransmitted after being multiplexed or packetized. For the transmissionof the background control information 13, a framework of thesupplemental enhancement information (SEI) in H.264/AVC or H.265/HEVCmay be used.

FIG. 3 shows a specific example of the background control informationgenerator 130. The background control information generator 130 shown inFIG. 3 includes a synchronous matcher 131 and an image correctionparameter generator 132.

The synchronous matcher 131 receives an input image 10, a fundamentalimage 11 and supplemental information 12. The synchronous matcher 131searches the fundamental image 11 for an image similar to the inputimage 10, based on the supplemental information 12, and generatesselection information representing at least one image searched for(which image is the same as the selection image described above). Theselection image may include, for example, an image which is included inthe fundamental image 11 and which most resembles the input image 10.The synchronous matcher 131 supplies the selection information and theselection image to the image correction parameter generator 132.

The image correction parameter generator 132 receives the input image10, and also receives the selection information and selection image fromthe synchronous matcher 131. The image correction parameter generator132 generates the correction parameter described above, based on theinput image 10 and the selection image. The image correction parametergenerator 132 generates background control information 13, includingselection information and a correction parameter, and supplies thegenerated background control information 13 to the background imagegenerator 120. In addition, the image correction parameter generator 132supplies the background control information 13 to the image decodingapparatus 200.

The image encoder 210 receives encoded data 15 transmitted from theimage encoding apparatus 100, and also receives a background image(signal) 17, to be mentioned later, from the background image generator220. The image decoder 210 decodes the encoded data 15, using thebackground image 17, thereby generating a reproduced image (signal) 18.The background image 17 is the same as (common to) the background image14 generated by the image encoding apparatus 100. The reproduced image18 is substantially the same as the input image. The image decoder 210supplies the reproduced image 18 to an external apparatus (e.g., adisplay not shown).

FIG. 5 shows a specific example of the image decoder 210. The imagedecoder 210 shown in FIG. 5 includes an entropy decoder 401, ade-quantizer/inverse transformer 402, an adder 403, a loop filter 404, adecoded image buffer 405, an external image buffer 406, and a predictedimage generator 407.

The entropy decoder 401 receives encoded data 15 transmitted from theimage encoding apparatus 100. The entropy decoder 401 executes entropydecoding for the encoded data 15, thereby obtaining the originalquantized transform coefficients and prediction information (includingmotion vector information, for example). The entropy decoder 401supplies the quantized transform coefficients to thede-quantizer/inverse transformer 402 and the prediction information tothe predicted image generator 407.

The de-quantizer/inverse transformer 402 receives the quantizedtransform coefficients from the entropy decoder 401. Thede-quantizer/inverse transformer 402 de-quantizes the quantizedtransform coefficients in accordance with the quantization parametermentioned above, thereby restoring the transform coefficients, andperforms inverse transform with respect to the transform coefficients(the inverse transform is processing just the opposite to the transformperformed by the image encoder 110), thereby restoring the predictionerror (signal). Inverse orthogonal transforms, such as an IDCT, are usedfor the inverse transforms of the transform coefficients. Thede-quantizer/inverse transformer 402 supplies the prediction error tothe adder 403.

The adder 403 receives the prediction error from thede-quantizer/inverse transformer 402, and the predicted image (signal)from the predicted image generator 407. The adder 403 adds theprediction error and the predicted image together, thereby generating adecoded image (signal). The adder 403 supplies the decoded image to theloop filter 404.

The loop filter 404 receives the decoded image from the adder 403. Theloop filter 404 applies loop filter processing to the decoded image. Forexample, the loop filter processing may be deblocking filter processing.The loop filter 404 supplies the filtered decoded image to the decodedimage buffer 405.

The decoded image buffer 405 receives the filtered decoded image fromthe loop filter 404. The decoded image buffer 405 stores the filtereddecoded image as a reference image. The decoded image buffer 405supplies the reference images to the predicted image generator 407, asneeded. The decoded image buffer 405 supplies the reference image 18 toan external apparatus (e.g., a display not shown) as a reproduced image18 in display order.

The external image buffer 406 receives the background image 17 from thebackground image generator 220. The external image buffer 406 stores thebackground image 17 as a reference image. The external image buffer 406supplies the reference images to the predicted image generator 407, asneeded.

The predicted image generator 407 receives the reference images(filtered decoded images) from the decoded image buffer 405, receivesthe reference image (background image 17) from the external image buffer406, and receives prediction information from the entropy decoder 401.The predicted image generator 407 generates a predicted image, usingboth the prediction information and the reference images. For example,the predicted image generator 407 generates a predicted image byperforming intra prediction, motion compensation prediction for thereference images, or prediction from the background image 17(corresponding to the inter-layer prediction in hierarchical coding).The predicted image generator 407 supplies the predicted image to theadder 403.

The background image generator 220 receives a fundamental image (signal)16 and further receives background control information 13 transmittedfrom the image encoding apparatus 100. The background image generator220 may have a configuration identical or similar to that of backgroundimage generator 120. The background image 16 is the same as (common to)the background image 11 used by the image encoding apparatus 100. Thebackground image generator 220 generates a background image 17 based onthe background control information 13 and the fundamental image 16. Thebackground image generator 220 supplies the background image 17 to theimage decoder 210.

As described above, in the image transmission system of the firstembodiment, the image encoding apparatus generates background controlinformation based on the input image and transmits it to the imagedecoding apparatus. In addition, the image encoding apparatus generatesa background image based on this background control information and afundamental image prepared beforehand, encodes this input image usingthe background image, and transmits the encoded data to the imagedecoding apparatus. The image decoding apparatus generates a backgroundimage (which is the same image as used by the image encoding apparatus)by using the transmitted background control information and thefundamental image (which is the same image as used by the image encodingapparatus), and decodes the encoded data using the background image.Accordingly, the image transmission system of the embodiment does nothave to encode or transmit most of the background region included in theinput image, and therefore significantly reduces the amount of dataencoded. That is, images of high quality can be transmitted even if thebandwidth of a channel is narrow.

Second Embodiment

As shown in FIG. 6, the image transmission system of the secondembodiment comprises an image encoding apparatus 500 and an imagedecoding apparatus 600. The image encoding apparatus 500 includes animage encoder 110 and a background image generator 120. The imagedecoding apparatus 600 includes an image decoder 210, a background imagegenerator 220 and a background control information generator 230. Theimage transmission system shown in FIG. 6 may be, for example, a systemthat transmits images photographed by a monitor camera to a remoteoperator terminal.

The image encoder 110 shown in FIG. 6 differs from the image encoder 110shown in FIG. 1 in that it transmits encoded data 15 to the imagedecoding apparatus 600. The encoded data 15 may be transmitted to theimage decoding apparatus 600 by wireless communications or suppliedthereto by wire communications; alternatively, the encoded data 13 maybe transferred to the image decoding apparatus 600, using a recordingmedium. The background image generator 120 shown in FIG. 6 differs fromthe background image generator 120 in that it receives backgroundcontrol information 13 transmitted from the image decoding apparatus600.

The image decoder 210 shown in FIG. 6 differs from the image decoder 210shown in FIG. 1 in that it receives encoded data 15 transmitted from theimage decoding apparatus 500. The image decoder 210 supplies thereproduced image 18 not only to an external apparatus (e.g., a displaynot shown) but also to the background control information generator 230.The background image generator 220 shown in FIG. 6 differs from thebackground image generator 220 shown in FIG. 1 in that it receivesbackground control information 13 transmitted from the backgroundcontrol information generator 230.

The background control information generator 230 receives a fundamentalimage 16 and supplemental information 19, and receives reproduced image18 from the image decoder 210. The background control informationgenerator 230 searches the fundamental image 16 for an image similar tothe reproduced image 18, based for example on the supplementalinformation 19, and generates background control information 13including selection information representing the image searched for(which image is the same as the selection image selected as the base ofthe background image 17). Furthermore, the background controlinformation generator 230 may generate a correction parameter used forcorrecting the background image 17 and include it in the backgroundcontrol information 13.

In summary, the selection information described in relation to the firstembodiment represents a selection image that is similar to the inputimage 10 to be encoded, while the selection information described inrelation to the second embodiment represents a selection image that issimilar to an reproduced image 18 (not the input image 10 to be encoded)reproduced by decoding a previously-encoded input image. Therefore, thesecond embodiment is suitably applied to the case where the backgroundregion included in the input image 10 does not undergo a significanttemporal change.

The supplemental information 19 may include information representing,for example, a photographing position, a photographing direction, aphotographing date, a photographing time, etc. These kinds ofinformation are useful when the background control information generator230 searches the fundamental image 16 for an image similar to thereproduced image 18. For example, the information on the photographingposition and photographing direction is useful when the fundamentalimage 16 is searched for an image which is similar to the reproducedimage 10 in terms of an object (especially, a background) to bephotographed. The information on the photographing date and time isuseful when the fundamental image 16 is searched for an image which issimilar to the reproduced image 18 in terms of the sunshine condition.

The background control information generator 230 may search thefundamental image 16 for an image similar to the reproduced image 18,without using the supplemental information 19. Alternatively, thebackground control information generator 230 may select at least oneimage from the fundamental image 16, without reference to the reproducedimage 18. In order to improve the coding rate, an image similar to thereproduced image 18 should be preferably searched for. In order toimprove the search efficiency, the supplemental information 19 should bepreferably used.

The background control information generator 230 supplies the backgroundcontrol information 13 to the background image generator 220. Inaddition, the background control information generator 230 supplies thebackground control information 13 to the image encoding apparatus 500.The background control information 13 may be transmitted to the imageencoding apparatus 500 by wireless communications or supplied thereto bywire communications; alternatively, the background control information13 may be transferred to the image encoding apparatus 500, using arecording medium.

FIG. 7 shows a specific example of the background control informationgenerator 230. The background control information generator 230 shown inFIG. 7 includes a predictive matcher 231 and an image correctionparameter generator 232.

The predictive matcher 231 receives a fundamental image 16 andsupplemental information 19, and receives reproduced image 18 from theimage decoder 210. The predictive matcher 231 searches the fundamentalimage 16 for an image similar to the reproduced image 18, based on thesupplemental information 19, and generates selection informationrepresenting at least one image searched for (which image is the same asthe selection image described above). The selection image may comprise,for example, an image which is included in the fundamental image 16 andwhich most resembles the reproduced image 18. The predictive matcher 231supplies the selection information and the selection image to the imagecorrection parameter generator 232.

The image correction parameter generator 232 receives the reproducedimage 18 from the image decoder 210, and also receives the selectioninformation and selection image from the predictive matcher 231. Theimage correction parameter generator 232 generates the correctionparameter described above, based on the reproduced image 18 and theselection image. The image correction parameter generator 232 generatesbackground control information 13, including selection information and acorrection parameter, and supplies the generated background controlinformation 13 to the background image generator 220. In addition, theimage correction parameter generator 232 supplies the background controlinformation 13 to the image encoding apparatus 500.

As described above, in the image transmission system of the secondembodiment, the image decoding apparatus generates background controlinformation based on the decoded reproduced image and transmits it tothe image encoding apparatus. In addition, the image encoding apparatusgenerates a background image based on transmitted background controlinformation and a fundamental image prepared beforehand, encodes theinput image using the background image, and transmits the encoded datato the image decoding apparatus. The image decoding apparatus generatesa background image (which is the same image as used by the imageencoding apparatus), based on the generated background controlinformation and the fundamental image (which is the same image as usedby the image encoding apparatus), and decodes the encoded data using thebackground image. Accordingly, the image transmission system of theembodiment does not have to encode or transmit most of the backgroundregion included in the input image, and therefore significantly reducesthe amount of data encoded. That is, images of high quality can betransmitted even if the bandwidth of a channel is narrow.

Third Embodiment

As shown in FIG. 8, the image transmission system of the thirdembodiment comprises an image encoding apparatus 500, an image decodingapparatus 200 and a background control information generation apparatus700. The background control information generation apparatus 700includes a background control information generator 710. The imagetransmission system shown in FIG. 8 may be, for example, a system thattransmits images photographed by a monitor camera to a remote operatorterminal.

The background image generator 120 shown in FIG. 8 differs from thebackground image generator 120 in that it receives background controlinformation 13 transmitted from the background control informationgeneration apparatus 700. The background image generator 220 shown inFIG. 8 differs from the background image generator 220 shown in FIG. 1in that it receives background control information 13 transmitted fromthe background control information generation apparatus 700.

The background control information generator 710 receives a fundamentalimage 20 and supplemental information 21. The background controlinformation generator 710 searches the fundamental image 20 and selectsat least one image based on the supplemental information 21, andgenerates background control information 13 including selectioninformation representing the selected image. The fundamental image 20 isthe same as (common to) the fundamental image 11 used by the imageencoding apparatus 500 or the fundamental image 16 used by the imagedecoding apparatus 200. The supplemental information 21 may be the sameas or similar to the aforementioned supplemental information 12 or theaforementioned supplemental information 19.

In addition, the background control information generator 710 suppliesthe background control information 13 to the image encoding apparatus500 and the image decoding apparatus 200. The background controlinformation 13 may be transmitted to the image encoding apparatus 500and the image decoding apparatus 200 by wireless communications orsupplied thereto by wire communications; alternatively, the backgroundcontrol information 13 may be transferred to the image encodingapparatus 500 and the image decoding apparatus 200, using a recordingmedium.

The background control information generator 710 may be designed in themanner shown in FIG. 9. The background control information generator 710shown in FIG. 9 includes an image selector 711. The image selector 711receives a fundamental image 20 and supplemental information 21. Theimage selector 711 searches the fundamental image 20 and selects atleast one image based on the supplemental information 21, and generatesbackground control information 13 including selection informationrepresenting the selected image. The image selector 711 supplies thebackground control information 13 to the image encoding apparatus 500and the image decoding apparatus 200.

As described above, in the image transmission system of the thirdembodiment, the background control information generation apparatusgenerates background control information and transmits it to the imageencoding apparatus and the image decoding apparatus. In addition, theimage encoding apparatus generates a background image based ontransmitted background control information and a fundamental imageprepared beforehand, encodes the input image using the background image,and transmits the encoded data to the image decoding apparatus. Theimage decoding apparatus generates a background image (which is the sameimage as used by the image encoding apparatus) by using the transmittedbackground control information and the fundamental image (which is thesame image as used by the image encoding apparatus), and decodes theencoded data using the background image. Accordingly, the imagetransmission system of the embodiment does not have to encode ortransmit most of the background region included in the input image, andtherefore significantly reduces the amount of data encoded. That is,images of high quality can be transmitted even if the bandwidth of achannel is narrow.

Fourth Embodiment

As shown in FIG. 10, the image transmission system of the fourthembodiment comprises an image encoding apparatus 800 and an imagedecoding apparatus 900. The image encoding apparatus 800 includes animage encoder 110, a background image generator 820 and a backgroundcontrol information generator 830. The image decoding apparatus 900includes an image decoder 210 and a background image generator 920. Theimage transmission system shown in FIG. 10 may be, for example, a systemthat transmits images photographed by a monitor camera to a remoteoperator terminal.

The image encoder 110 shown in FIG. 10 differs from the image encoders110 shown in FIGS. 1, 6 and 8 in that it receives background image 14from the background image generator 820 and in that it transmits encodeddata 15 to the image decoding apparatus 900. In addition, the imageencoder 110 supplies the reproduced image (signal) 22 to the backgroundimage generator 820. The reproduced image 22 may be either a localdecoded image or an image obtained by executing some kind of imageprocessing (e.g., filtering processing) for the local decoded image.

The image decoder 210 shown in FIG. 10 differs from the image decoders210 shown in FIGS. 1, 6 and 8 in that it receives background image 17from the background image generator 920 and in that it receives encodeddata 15 transmitted from the image encoding apparatus 800. In addition,the image decoder 210 supplies the reproduced image 18 not only to anexternal apparatus (e.g., a display not shown) but also to thebackground image generator 920.

The background image generator 820 receives a reproduced image 22 fromthe image encoder 110, and also receives background control information13 from the background control information generator 830. The backgroundimage generator 820 generates a background image 14 based on thebackground control information 13 and the reproduced image 22. Thebackground image generator 820 supplies the background image 14 to theimage encoder 110. That is, the background image generator 820 generatesa background image 14, using the reproduced image 22 in place of thefundamental image 11 described above.

As described above, the background control information 13 includes aparameter required for generating the background image 14. To bespecific, the background control information 13 shown in FIG. 10 mayinclude selection information representing at least one image (selectionimage) that is selected as the base of the background image 14 of atleast one reproduced image 22. The selection information may be a framenumber if the reproduced image 22 is a moving picture. The backgroundcontrol information 13 includes a correction parameter used forenhancing the quality of the background image 14. The background imagegenerator 820 may be designed in such a manner that the backgroundcontrol information 13 does not include selection information. Forexample, the background image generator 820 may be designed toautomatically select a predetermined number of recently reproducedimages.

For example, the background image generator 820 may generate thebackground image 14 by selecting one image (selection image) from atleast one reproduced image 22. Alternatively, the background imagegenerator 820 may generate the background image 14 by selecting aplurality of images (selection images) from a plurality of reproducedimages 22 and executing background pixel calculation (to be describedlater), using the selection images. In addition, the background imagegenerator 820 may execute format conversion or image correction for thebackground image 14 generated as above, before the background image 14is supplied to the image encoder 110.

If the background image 14 differs from the input image 10 in format,then the background image generator 820 converts the format of thebackground image 14 to the same format as the input image 10. The formatis, for example, an image size, an image bit depth, a color differenceformat, a color space, etc.

Where the background control information 13 includes a correctionparameter, the background image generator 820 corrects the backgroundimage 14 using the correction parameter. To be more specific, thebackground image generator 820 may execute gamma correction, using thegamma value included in the correction parameter. The background imagegenerator 820 may execute histogram conversion, using the histograminformation on the input image included in the correction parameter. Thebackground image generator 820 may execute Wiener filter processing forthe background image 14, using the filter coefficient of the Wienerfilter included in the correction parameter (the filter coefficient canbe determined based on the input image 10 and the background image 14).The background image generator 820 may execute weighted prediction ofthe background image 14, for example, for the correction of thebrightness, using the weight coefficient and offset value included inthe correction parameter. The background image generator 820 may executegeometric transform of the background image 14, for example, for thecorrection of the photographing position, using the geometric-transformparameter value included in the correction parameter. The backgroundimage generator 820 may execute a variety of image corrections notmentioned herein.

FIG. 11 shows a specific example of the background image generator 820.The background image generator 820 shown in FIG. 11 includes areproduced image buffer 821, a background pixel calculator 822 and abackground image buffer 123. The background image generator 123 shown inFIG. 11 differs from the background image generator 123 shown in FIG. 2in that it receives a background image 14 supplied from the backgroundpixel calculator 822.

The reproduced image buffer 821 receives a reproduced image 22 from theimage encoder 110 and stores it therein. The reproduced image 22 storedin the reproduced image buffer 821 is read out by the background pixelcalculator 822, as needed.

The background pixel calculator 822 receives background controlinformation 13 from the background control information generator 830.The background pixel calculator 822 selects a plurality of images from aplurality of reproduced images 22 and reads the selected images from thereproduced image buffer 821. The background pixel calculator 822generates a background image 14 (pixel values constituting thebackground image 14) by executing background pixel calculation using theselection images read out from the reproduced image buffer 821. Thebackground pixel calculator 822 supplies the background image 14 to thebackground image buffer 123.

The background pixel calculation may be any kind of arithmeticprocessing as long as background pixels can be derived from a pluralityof images (selection images) that are adjusted in position andphotographed at different photographing times. For example, thebackground pixel calculator 822 may calculate a mode of the pixel valuesat predetermined positions of the selection images, thereby deriving thepixel values of the predetermined position of the background image 14.The background pixel calculator 822 may calculate a median or an averagein place of the mode. The background pixel calculator 822 may determinea threshold used for determining non-background pixels (foregroundpixels), based on the average and variance of the pixel values atpredetermined positions of a plurality of selection values. In thiscase, the background pixel calculator 122 first excludes non-backgroundpixels from the pixel values, and re-calculates an average to derive thepixel values at the predetermined position of the background image 14.Alternatively, the background pixel calculator 822 may calculate each ofthe pixel values of the background image 14, using the Gaussian mixturemodel (GMM) or the Godbehere, Matsukawa, Goldberg algorithm (GMGalgorithm). The background pixel calculator 822 may calculate each ofthe pixel values of the background image 14, using various encodingparameters regarding the reproduced image 22 (e.g., skipping informationfor motion compensation, or an absolute value sum of prediction errors).

A general image transmission system uses a color space in which a pixelvalue is expressed based on one brightness signal and twocolor-difference signals. However, the brightness signal is easilyaffected by a shadow in the background pixel calculation. If the colorspace of a selection image includes a brightness signal, therefore, thebackground pixel calculator 822 may first convert the color space into acolor space including no brightness signal (e.g., RGB) and then executebackground pixel calculation. Furthermore, the background pixelcalculator 822 may change each pixel value of the background image 14back into an original color space, as needed.

The background control information generator 830 receives an input image10 and supplemental information 12. For example, the background controlinformation generator 830 checks the reproduced image 22 and predicts animage which is very likely to resemble the input image 10, based on thesupplemental information 12, and generates background controlinformation 13 including selection information representing thepredicted image (which image is the same as the selection imagedescribed above). Furthermore, the background control informationgenerator 830 may generate a correction parameter used for correctingthe background image 14 and include it in the background controlinformation 13.

The background control information generator 830 may check thereproduced image 22 and predict an image which is very likely toresemble the input image, without using the supplemental information 12.Alternatively, the background control information generator 830 mayselect at least one image from the reproduced image 22, withoutreference to the input image 10. In order to improve the coding rate, animage similar to the input image 10 should be preferably predicted. Inorder to improve the prediction efficiency, the supplemental information12 should be preferably used.

The background control information generator 830 supplies the backgroundcontrol information 13 to the background image generator 820. Inaddition, the background control information generator 830 supplies thebackground control information 13 to the image decoding apparatus 900.The background control information 13 may be transmitted to the imagedecoding apparatus 900 by wireless communications or supplied thereto bywire communications; alternatively, the background control information13 may be transferred to the image decoding apparatus 900, using arecording medium.

The background control information 13 may be transmitted separately fromthe encoded data 15 or together with the encoded data 15. For example,the background control information 13 and the encoded data 15 aretransmitted after being multiplexed or packetized. For the transmissionof the background control information 13, a framework of the SEI inH.264/AVC or H.265/HEVC may be used.

FIG. 12 shows a specific example of the background control informationgenerator 830. The background control information generator 830 shown inFIG. 12 includes an image correction parameter generator 831.

The image correction parameter generator 831 receives an input image 10and supplemental information 12. The image correction parametergenerator 831 generates the correction parameter described above, basedon the input image 10 and the supplemental information 12. The imagecorrection parameter generator 831 generates background controlinformation 13, including a correction parameter, and supplies thegenerated background control information 13 to the background imagegenerator 820. In addition, the image correction parameter generator 831supplies the background control information 13 to the image decodingapparatus 900.

The background image generator 920 receives a reproduced image 18 andfurther receives background control information 13 transmitted from theimage encoding apparatus 800. The background image generator 920 mayhave a configuration identical or similar to that of background imagegenerator 820. The reproduced image 18 is the same as the reproducedimage 22 used by the image encoding apparatus 800. The background imagegenerator 920 generates a background image 17 based on the backgroundcontrol information 13 and the reproduced image 18. The background imagegenerator 920 supplies the background image 17 to the image decoder 210.In summary, the background image generator 920 generates a backgroundimage 17, using the reproduced image 18 in place of the fundamentalimage 16 described above.

As described above, in the image transmission system of the fourthembodiment, the image encoding apparatus generates background controlinformation based on the input image and transmits it to the imagedecoding apparatus. In addition, the image encoding apparatus generatesa background image based on this background control information and areproduced image corresponding to the local decoded image of the encodedinput image, encodes this input image using the background image, andtransmits the encoded data to the image decoding apparatus. The imagedecoding apparatus generates a background image (which is the same imageas used by the image encoding apparatus) using both the transmittedbackground control information and the decoded reproduced image, anddecodes the encoded data using the background image. Accordingly, theimage transmission system of the embodiment does not have to encode ortransmit most of the background region included in the input image, andtherefore significantly reduces the amount of data encoded. That is,images of high quality can be transmitted even if the bandwidth of achannel is narrow.

The image encoding apparatus, image decoding apparatus and backgroundcontrol information generation apparatus included in the imagetransmission system of each embodiment may be implemented by thehardware components shown in FIG. 13. The hardware components shown inFIG. 13 include a central processing unit (CPU) 1001, a read only memory(ROM) 1002, a random access memory (RAM) 1003, a communication interface(IF) 1004 and a bus 1005.

Data is exchanged between the CPU 1001, ROM 1002, RAM 1003 andcommunication IF 1004 through the bus 1005.

The CPU 1001 reads a program from a recording medium (e.g., ROM 1002)and stores it in the main memory. By executing the program, the CPU 1001operates as functional units for performing the operations describedabove (e.g., as a predicted image generator 308). The RAM 1003 may beused as various buffers described above. The communication IF 1004 maybe used for transmitting/receiving the background control information13, encoded data 15, etc.

At least a part of the processing in the above-described embodiments canbe implemented using a general-purpose computer as basic hardware. Aprogram implementing the processing in each of the above-describedembodiments may be stored in a computer readable storage medium forprovision. The program is stored in the storage medium as a file in aninstallable or executable format. The storage medium is a magnetic disk,an optical disc (CD-ROM, CD-R, DVD, or the like), a magnetooptic disc(MO or the like), a semiconductor memory, or the like. That is, thestorage medium may be in any format provided that a program can bestored in the storage medium and that a computer can read the programfrom the storage medium. Furthermore, the program implementing theprocessing in each of the above-described embodiments may be stored on acomputer (server) connected to a network such as the Internet so as tobe downloaded into a computer (client) via the network.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel methods and systems describedherein may be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the methods andsystems described herein may be made without departing from the spiritof the inventions. The accompanying claims and their equivalents areintended to cover such forms or modifications as would fall within thescope and spirit of the inventions.

What is claimed is:
 1. An image encoding apparatus comprising: abackground control information generator that generates backgroundcontrol information based on an input image, the background controlinformation used for generating a first background image from at leastone first image; a background image generator that generates the firstbackground image, based on the background control information and atleast one first image; and an image encoder that encodes the input imageusing the first background image to generate encoded data.
 2. Theapparatus according to claim 1, wherein the at least one first imageincludes at least one fundamental image prepared beforehand.
 3. Theapparatus according to claim 2, wherein the background controlinformation includes selection information representing an imageselected for generating the first background image from the at least onefundamental image.
 4. The apparatus according to claim 2, wherein thebackground control information generator searches the at least onefundamental image for an image that most resembles the input image andgenerates background control information including selection informationthat represents the image searched for.
 5. The apparatus according toclaim 4, wherein the background control information generator searchesthe at least one fundamental image for an image that most resembles theinput image, based on the supplemental information.
 6. The apparatusaccording to claim 5, wherein the supplemental information includesinformation representing at least one of a photographing date, aphotographing time, a photographing position and a photographingdirection.
 7. The apparatus according to claim 1, wherein the inputimage is an image photographed by a camera attached to a first movingobject when the first moving object travels along a predetermined route,and the at least one fundamental image is a past image which isphotographed by a camera attached to a second moving object when thesecond moving object travels along the route.
 8. The apparatus accordingto claim 1, wherein the background control information includes acorrection parameter used for correcting the first background image. 9.The apparatus according to claim 1, wherein the at least one first imageincludes a reproduced image corresponding to a local decoded image of anencoded input image.
 10. The apparatus according to claim 1, wherein theimage encoder generates the encoded data by performing predictiveencoding for the input image, using a reference image including thefirst background image.
 11. An image decoding apparatus comprising: abackground control information generator that generates backgroundcontrol information based on a previously decoded image, the backgroundcontrol information used for generating a background image from at leastone image; a background image generator that generates the backgroundimage, based on the background control information and at least oneimage; and an image decoder that decodes encoded data using thebackground image to generate a reproduced image.
 12. An imagetransmission method comprising: generating background controlinformation; transmitting the background control information to at leastone of an image encoding apparatus and an image decoding apparatus;causing the image encoding apparatus to generate a first backgroundimage based on the background control information and at least one firstimage; causing the image encoding apparatus to encode an input imageusing the first background image to generate encoded data; causing theimage encoding apparatus to transmit the encoded data to the imagedecoding apparatus; causing the image decoding apparatus to generate asecond background image common to the first background image, based onthe background control information and at least one second image that iscommon to the at least one first image, and causing the image decodingapparatus to decode the encoded data using the second background imageto generate a reproduced image.